1. Field of the Invention
The present invention relates generally to a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks, and a hybrid error concealment method, and more particularly, to a method of using spatial redundancy information and temporal redundancy information for concealing a spatial image error of an encoded image frame caused by missing macroblocks.
2. The Prior Arts
Image transmission service including video meeting, website browsing, image file transmission, is a very important service provided by communication and IT enterprises. Typically, an original image file desired to be transmitted is often too large to be effectively transmitted over the internet, or often occupies too much memory space. As such, an image file is often performed with a coding process with a high compression ratio and a low loss ratio before transmission for reducing the size of the image content. Such a coding process may be selected from TIFF, JPEG, MJEP, MPEG, H.264/AVC, in accordance with the demands of the system for the image compression ratio, the loss ratio, so as to obtain an optimal algorithm for static images or continuous images.
However, when being transmitted, the encoded image file is often interfered by the transmission interface or the electrical system of the receiving end. Such interferences usually cause damages or errors of the image file content, so that the receiving end becomes incapable of decoding and recovering the encoded image file back to original by a predetermined corresponding decoding method.
In accordance with psychology of vision, an individual edge would not affect visional recognition. However, such an individual edge may be mutually affected with edges adjacent thereto so as to generate a composition effect. In this concern, human vision is more sensitive to edge distortions. Generally, edge distortions include blurred edges and newly created false edges. As such, it is very important to maintain original edges and avoid the creation of false edges when executing an image process.
A method of recovering or obtaining image content of a missing macroblock according to image content of correct macroblocks of a same frame of the missing macroblock is generally called the SEC method. A further method proposed as another solution is called the temporal error concealment (TEC) method. According to the TEC method, a motion vector of a missing macroblock can be achieved by referring to a correct image data of a previous frame. Then, a reference frame is moved according to the motion vector, and therefore the image data of the missing macroblock can be thus obtained. In such a way, the negative affection to the entire image frame caused by the missing macroblock can be concealed.
The conventional SEC method is schematically introduced firstly as below.
There are many algorithms of conventional technologies have been proposed for spatial error concealment, such as bilinear interpolation (BI), directional interpolation (DI), and best neighborhood matching (BNM). The principles of the aforementioned algorithms can be learnt in more details by referring to related references, such as: P. Salama, N. B. Shroff, and E. J. Delp, “Error concealment in encoded video streams,” in Signal Recovery Techniques for Image and Video Compression and Transmission, A. K. Katsaggelos and N. P. Galatsanos, Eds. Norwell, Mass.: Kluwer, ch. 7, 1998; H. Sun and W. Kwok, “Concealment of damaged block transform coded images using projection onto convex set,” IEEE Trans. Image Processing, vol. 4, pp. 470-477, April 1995; and Z. Wang, Y. Yu, and D. Zhang, “Best neighborhood matching: An information loss restoration technique for block-based image coding systems,” IEEE Trans. Image Process., vol. 7, no. 7, pp. 1056-1061, July 1998. They are to be briefly illustrated herebelow for better understanding of the present invention.
FIG. 1 is a schematic diagram illustrating a BI method according to a conventional technology. Referring to FIG. 1, a missing macroblock 20 having no correct image content is surrounded by edge reference pixels 21 containing correct image content. A missing pixel P(x, y) in the missing macroblock 20 positioned at the coordinates (x, y) of BI is then interpolated by the formula (1), according to the correct image content of the four edges in both horizontal direction and vertical direction:
                                          p            ⁡                          (                              x                ,                y                            )                                =                                                    p                ⁢                                                                  ⁢                1                ×                d                ⁢                                                                  ⁢                2                            +                              p                ⁢                                                                  ⁢                2                ×                d                ⁢                                                                  ⁢                1                            +                              p                ⁢                                                                  ⁢                3                ×                d                ⁢                                                                  ⁢                4                            +                              p                ⁢                                                                  ⁢                4                ×                d                ⁢                                                                  ⁢                3                                                                    d                ⁢                                                                  ⁢                1                            +                              d                ⁢                                                                  ⁢                2                            +                              d                ⁢                                                                  ⁢                3                            +                              d                ⁢                                                                  ⁢                4                                                    ,                            (        1        )            where d1, d2, d3, d4 are relative distances from the edge pixels P1, P2, P3, P4 to the missing pixel P(x, y), respectively.
FIG. 2 is a schematic diagram illustrating a DI method according to a conventional technology. As shown in FIG. 2, the edge pixels contain correct image content, while the macroblock surrounded by the edge pixels is a missing macroblock containing incorrect image content. The pixel P(x, y) in the missing macroblock positioned at the coordinates (x, y) is then interpolated by the formula (2), according to the correct image content of two edges along a specific direction:
                                          p            ⁢                          (                              x                ,                y                            )                                =                                                    p                ⁢                                                                  ⁢                1                ×                d                ⁢                                                                  ⁢                1                            +                              p                ⁢                                                                  ⁢                2                ×                d                ⁢                                                                  ⁢                2                                                                    d                ⁢                                                                  ⁢                1                            +                              d                ⁢                                                                  ⁢                2                                                    ,                            (        2        )            where d1, d2 are relative distances from the edge pixels P1, P2, to the missing pixel P(x, y), respectively.
FIG. 3 is a directional schematic diagram illustrating a DI method according to a conventional technology. As shown in FIG. 3, in the DI method, the edge direction is classified into 8 directions, in which DI(0°) represents a direction of 0°, DI(22.5°) represents a direction of 22.5°, . . . and so forth. P1 and P2 of FIG. 2 are each of one of the 8 directions.
The conventional DI method usually employs a direction filter of an edge direction detection technique for determining direction of each pixel contained in the macroblocks surrounding the missing macroblock. Sobel operator or Prewitt operator are often used. Sobel operator, which is also known as a Sobel filter, is to be exemplified for illustration below taking a 3×3 Sobel operator as an example:
                                          S            x                    =                      [                                                                                -                    1                                                                    0                                                  1                                                                                                  -                    2                                                                    0                                                  2                                                                                                  -                    1                                                                    0                                                  1                                                      ]                          ⁢                                  ⁢                              S            y                    =                      [                                                                                -                    1                                                                                        -                    2                                                                                        -                    1                                                                                                0                                                  0                                                  0                                                                              1                                                  2                                                  1                                                      ]                                              (        3        )            where Sx and Sy detect edge characteristics at X direction and Y direction, respectively, and obtain directional gradients at X direction and Y direction of equation (4) (Gx and Gy), respectively.GX=Xi+1,j−1−Xi−1,j−1+2Xi+1,j−2Xi−1,j+Xi+1,j+1−Xi−1,j+1 Gy=Xi−1,j+1−Xi−1,j−1+2Xi,j+1−2Xi,j−1+Xi+1,j+1−Xi−1,j−1  (4).
A directional gradient (Grand) and a directional angle (θ) can be further calculated with equations (5) and (6).Grand=√{square root over (Gx2+Gy2)}  (5)θ=tan−1(Gx/Gy)  (6)where the directional gradient (Grand) is a quantity factor for evaluating the strength of the directivity. In other words, a directional angle (θ) having a maximum directional gradient (Grand) is selected serving as a direction reference for the missing macroblock.
If all of the directional gradients (Grand) are too low, or even lower than a threshold, the missing macroblock is determined as having no directivity, and not adapted for SEC by DI method. As such, other methods, (e.g., BI method), are demanded for processing those missing macroblocks having no directivity.
FIG. 4 is a schematic diagram illustrating a BNM method according to a conventional technology. Referring to FIG. 4, a missing macroblock 20 of frame 40 is surrounded by a plurality of reference macroblocks 22, and a best matching macroblock 32 is surrounded by a plurality of target macroblocks 30 in a searching area 42. The BNM method is for finding out the best matching macroblock 32 for replacing the missing macroblock 20. The BNM method includes the steps of: (1) taking N-pixel-wide boundary surrounding the missing macroblock as a searching image; (2) identifying best matching macroblocks nearest to the missing macroblock; and (3) replacing the corresponding missing macroblock (i.e., the macroblock surrounded by the best matching macroblocks) with the identified best matching macroblock.
Features of the BI method, ID method, and BNM method are to be discussed below in further details.
The DI method has the advantages of edge protection. For example, if there is only one strong edge near the missing macroblock, the original strong edge can be preserved and avoided from becoming blur. However, when there are several edges near the missing macroblock, the DI method may generate several false edges. Unfortunately, human vision is usually very sensitive to blur false edges. Correspondingly, the BI method is adapted to generate new blur false edges. However, if there is only one strong edge near the missing macroblock, the BI method may blur the original strong edge because the edges of the missing macroblock are interpolation processed and mixed with the adjacent pixels. Further, with respect to missing macroblocks having patterns of higher complexity, it is often difficult to obtain necessary effective reference blocks. Although the BNM method can recover the complex missing macroblock, it may cause edge discontinuity. It may also cause higher computation complexity than BI and DI methods, and thus is featured with a lower overall efficiency.
Further, a multi-direction interpolation (MDI) method has been proposed for preserving multi-directional edges. The MDI method is as discussed below.
FIG. 5 is a schematic diagram illustrating reference blocks of the MDI method. Referring to FIG. 5, the MDI selects a plurality of N-pixel-wide boundary reference blocks 50 as a reference block group surrounding a missing macroblock 20.
First, a Sobel operator is used to calculate a directional strength (i.e., directional gradient) and a directional angle of individual pixels of the 4×4 reference block as of the DI method. And then, a directional entropy (DE) Hd is calculated according to equation (7) for representing a directional factor.Hd=−Σp(dx)log2 p(dx)  (7),where dx represents a directional angle of a specific pixel of the reference block, while p(dx) represents a directional probability density function related to the directional angle.
A greater DE indicates a smaller directivity (i.e., having no specific edge direction), while a maximum DE value (i.e., 1) indicates absolutely evenly distributed edge directions.
Then, a direction threshold β is calculated according to equation (8), for obtaining an overall directional index of the 4×4 reference block. Thus the overall directional index can be accorded to determine whether to adopt the directional angle of the reference block as the directional angle of the missing macroblock group.
                              β          =                                    -                              ∑                                                                  ⁢                                                      p                    ⁡                                          (                                              d                        x                                            )                                                        ⁢                                      log                    2                                    ⁢                                      p                    ⁡                                          (                                              d                        x                                            )                                                                                                          Max              ⁡                              (                                  H                  d                                )                                                    ,                            (        8        )            where Max(Hd) represents a maximum value of Hd.
Only when the direction threshold β is smaller than a specific threshold determining value, the directional angle is adopted, and when the direction threshold β is greater than the specific threshold determining value, the missing macroblock is determined as having no specific directivity. According to an aspect of the embodiment, the direction threshold β is preferably defined between 0.7 and 0.9. as such, the reference block group can be classified into four categories according to the direction threshold β, including no-directional background area, single-directional edge area, multi-directional edge area, and complex multi-directional edge area. With respect to reference blocks of different categories, different interpolation methods or matching methods are employed for recovering the missing macroblock. Specifically, the BI method is applied to the no-directional background area; the DI method is applied to the single-directional edge area; the BNM method is applied to the multi-directional edge area; and the MDI method is applied to the complex multi-directional edge area.
When the 4×4 reference block exhibits more than two strong edge directions, the missing macroblock is considered as a multi-directional edge area. In this case, the missing macroblock is recovered by calculating missing pixels of the missing macroblock by interpolation according to the directivity of the 4×4 reference block. According to an aspect of the embodiment, the MDI method is a 3-directional weighted interpolation method, compensating along a horizontal direction, a vertical direction, and a 45° direction. FIG. 6 is a schematic diagram illustrating a 3-directional weighting interpolation of the MDI method according to a conventional technology. Referring to FIG. 6, the missing pixels P are recovered according to the two horizontal pixels PH1 and PH2 at the horizontal direction, the two vertical pixels PV1 and PV2 at the vertical direction, and the two diagonal pixels PD1 and PD2 at the diagonal direction. If the missing pixels P do not have a corresponding strong edge direction, the missing macroblock can be interpolated along the same direction corresponding to that of the nearest missing pixel. In this concern, the MDI method is similar to the DI method.
When most 4×4 reference blocks exhibit more than two strong edge directions, while the direction threshold β is greater than the threshold determining value, the missing macroblock is classified as a complex multi-directional edge area. If the missing macroblock includes complex content, the BNM method may be an ideal one for reconstructing the block to achieve an optimal image quality. However, the BNM method also requires a very large amount of calculation and a very long calculation time. On another hand, although requiring for a smaller calculation amount, the BI method is adapted for maintaining to display an ordinary image quality only.
All above-discussed methods are categorized as ordinary conventional SEC methods. Below, a fast MDI method is to be specifically introduced in more details.
Generally, the MDI method often spends 40% to 90% of calculation time in searching for possible directional angle and directional gradient. However, related directional data of the missing macroblock of the transmitted image would have been recorded during the original coding process. As such, the recorded data contained in the original coding can be used for reducing the calculation amount and saving the calculation time. In H.264/AVC coding, there is a correlation between intro-predication methods and edge direction. For example, an eight-directional 4×4 prediction method and a three-directional 16×16 prediction method can be used for pre-estimating for the edge direction of real images. Intra prediction information of the H.264/AVC contains data related to the original macroblock direction. As such, the fast MDI method of the present invention employs another fast estimation method, for determining whether to use the directional data of the missing macroblock contained in the original coding, so as to execute a direct interpolation. In such a way, it can reduce the complexity of the process of edge direction detection by 70% to 95% on average, and thus drastically improving the decoding efficiency.
FIG. 7 is a flow chart illustrating a fast classification of a fast SEC method according to a conventional technology. Referring to FIG. 7, missing macroblocks are classified into two categories, flat region 53 and compound region 54, according to the intra prediction method information 51 of the original coding. Missing macroblocks in the flat region 53 are to be compensation processed by DI or BI method according to the intra prediction method information, while missing macroblocks in the compound region 54 are to be compensation processed by the aforementioned method.
With respect to the six neighbor blocks of each missing macroblock (i.e., top, top-right, left, right, bottom, and bottom-left), there are at least five adjacent macroblocks chosen via 16×16 prediction method. It does not need to use any Sobel operator to detect the edge direction for a smooth region of an image. On the contrary, each pixel of the missing macroblocks can be interpolated along a single direction (e.g., vertical, horizontal, or diagonal) or bi-direction (vertical-horizontal) determined by majority decision based on the intra 16×16 prediction method including method 0 (vertical), method 1 (horizontal), method 3 (plane), and method 2(DC), of corresponding surrounding macroblocks.
As to the compound region, in a 16×16 and 4×4 mixing prediction method of the reference macroblocks, four-pixel-wide boundary is disjoined into 4×4-pixel blocks as shown in FIG. 8. For the purpose of ascertaining dominant edges of each 4×4-pixel block, the intra method information and a small amount of the edge direction filters (i.e., Sobel filters) are employed). The calculation order of the Sobel filter to estimate the directional angles and directional gradients of an inner layer of pixels, p1, p2, p3, and p4, is depicted in FIG. 8. The calculation is stopped if a desired block is obtained. For each 4×4-pixel block, edge detection begins from p1 and p2. If at least one of p1 and p2 exhibits a strong edge in the same direction of the intra method information, the block is determined as the desired block. If none of p1 and p2 exhibits a strong edge, p3 and p4 are further calculated. As to the intra DC prediction method, the 4×4-pixel block is directly regarded as having no strong edge, and thus does not need a Sobel filter. The final stage is similar as the foregoing embodiment of SEC method of the present invention, in which the missing macroblocks can be classified into four categories, including: no-directional background area 55, single-directional edge area 56, multi-directional edge area 57, and complex multi-directional edge area 58. Different categories are adapted for different interpolation methods. Specifically, the BI method is applied to the no-directional background area; the DI method is applied to the single-directional edge area; the BNM method is applied to the multi-directional edge area; and the MDI method is applied to the complex multi-directional edge area.
All of the above-discussed methods are called conventional intra high-speed SEC methods.
And now, the conventional TEC methods are to be discussed herebelow.
In conventional TEC methods, a zero motion vector, neighboring motion vectors (motion vectors), an available motion vector of a spatially corresponding macroblock in a previous frame, or a median (or an average) of spatially adjacent motion vectors can be used. A TEC method called motion field interpolation method is recently proposed. According to the motion field interpolation method, a best motion vector is determined from the above candidate motion vectors by the boundary matching algorithm (BMA), which calculates the minimum sum of absolute differences (SAD) along the one-pixel-wide boundary between the missing macroblock and its candidate replacements in the H.264 JM decoder.
However, the conventional technology usually calculates an optimal motion vector by complex computation, which requires a large amount of computation and consumes much time, and negatively affects the overall decoding efficiency.
It is assumed that all missing macroblocks have been accurately detected, and all boundary blocks surrounding the missing macroblock contain correct image data including motion vectors. Referring to FIG. 9, there is shown a flow chart illustrating a conventional TEC method. The flow chart includes steps S100 through S180, to be discussed in more details herebelow.
First, at step S100, an optimal regression (OR) plane is set up according to motion vectors of the boundary blocks. An N-pixel-wide block group surrounding the missing macroblock includes a plurality of N×N pixel-wide blocks, in which N is a positive integer. As shown in the drawing, the embodiment is exemplified with N=4 for illustration. However, in other embodiments, N can be any other positive integers other than 4.
FIG. 10 is a schematic diagram illustrating boundary blocks according to a conventional technology. Referring to FIG. 10, it shows a missing macroblock 20 including 16 missing blocks, i.e., DB1, DB2, . . . , DB16. The missing blocks are surrounded by 16 boundary blocks, i.e., BB1, BB2, . . . , BB16. Further, each of the boundary blocks BB1, BB2, . . . , BB16, has a corresponding motion vectors, MV1 through MV16, respectively, for representing a temporal correlation with a previous frame.
For convenience of illustration, a coordinate system is set up in FIG. 10, in which a center of the missing macroblock 20 is set as an origin of coordinate, i.e., P0 (0, 0), and a pixel is defined as a unit of length. In this case, coordinates of the boundary blocks BB1 through BB16 are respectively:BB1(x1, y1)=(−6, 10)BB2(x2, y2)=(−2, 10)BB3(x3, y3)=(2, 10)BB4(x4, y4)=(6, 10)BB5(x5, y5)=(10, 6)BB6(x6, y6)=(10, 2)BB7(x7, y7)=(10, −2)BB8(x8, y8)=(10, −6)BB9(x9, y9)=(6, −10)BB10(x10, y10)=(2, −10)BB11(x11, y11)=(−2, −10)BB12(x12, y12)=(−6, −10)BB13(x13, y13)=(−10, −6)BB14(x14, y14)=(−10, −2)BB15(x15, y15)=(−10, 2)BB16(x16, y16)=(−10, 6);and the motion vectors thereof are marked at centers of the corresponding boundary blocks.
The OR plane is constructed by a multiple polynomial regression model of equation (9): Z(x,y)=c+a1x−a2x2+b1y+b2y2, where Z represents the motion vectors, and a1, a2, b1, b2, c are coefficients to be determined.
After regression analysis, an equation (10) as following can be obtained.
                                          Z            ⁡                          (                              x                ,                y                            )                                =                      [                                          1                106496                            ⁢                                                ∑                                      i                    =                    1                                    16                                ⁢                                                                  ⁢                                                      MV                    i                                    ·                                      (                                                                  27                        ⁢                                                                                                  ⁢                                                  x                          i                          2                                                                    +                                              25                        ⁢                                                                                                  ⁢                                                  y                          i                          2                                                                    -                      3120                                        )                                                                        ]                          ⁢                                  ⁢                              x            2                    +                                                                                   [                                                            1                      960                                        ⁢                                                                  ∑                                                  i                          =                          1                                                16                                            ⁢                                                                                          ⁢                                                                        MV                          i                                                ·                                                  x                          i                                                                                                      ]                                ⁢                x                            +                                                [                                                            1                      106496                                        ⁢                                                                  ∑                                                  i                          =                          1                                                16                                            ⁢                                                                                          ⁢                                                                        MV                          i                                                ·                                                  (                                                                                    25                              ⁢                                                                                                                          ⁢                                                              x                                i                                2                                                                                      +                                                          27                              ⁢                                                                                                                          ⁢                                                              y                                i                                2                                                                                      -                            3120                                                    )                                                                                                      ]                                ⁢                                  y                  2                                            +                                                [                                                                          ⁢                                                            1                      960                                        ⁢                                                                  ∑                                                  i                          =                          1                                                16                                            ⁢                                                                                          ⁢                                                                        MV                          i                                                ·                                                  y                          i                                                                                                      ]                                ⁢                y                            -                                                1                  512                                ⁢                                                      ∑                                          i                      =                      1                                        16                                    ⁢                                                                          ⁢                                                            MV                      i                                        ·                                          (                                                                        15                          ⁢                                                                                                          ⁢                                                      x                            i                            2                                                                          +                                                  15                          ⁢                                                                                                          ⁢                                                      y                            i                            2                                                                          -                        1832                                            )                                                                                                                              (        10        )            
A curved plane defined by the equation (10) is the desired OR plane. FIG. 11 is a schematic diagram illustrating boundary blocks according to a conventional technology. As shown in FIG. 11, at step S110, corresponding predicted spatial motion vectors of the missing blocks DB1 through DB16 can be obtained by the equation (10).
At step S120, temporal motion vectors are set up according to a previous image frame, i.e., taking motion vectors of the missing macroblock in the previous image frame at corresponding positions as high temporal correlated motion vectors.
At step S130, candidate motion vectors are constructed according to the boundary matching algorithm (BMA). As discussed above, the spatial motion vectors are obtained at step S110, the temporal VMs are obtained at step S120, and now at the current step, a quantitative difference between each spatial motion vector and each temporal motion vector can be estimated by the adjacent external boundary matching errors (AEBME). The motion vector having a least AEBME is selected as the candidate motion vector.
FIG. 12 is a schematic diagram illustrating a boundary matching algorithm (BMA) and related blocks according to a conventional technology. Referring to FIG. 12, the 16 missing blocks of the missing macroblock 20 are renamed with A00, A01, A110, A11, B00, B01, B10, B11, C00, C01, C10, C11, D00, D01, D10, D11, for the convenience of illustration.
The missing blocks A11, B11, C11, D11 are located at an inside layer of the missing macroblock 20, and are not neighboring to the boundary blocks having correct image data. The rest missing blocks locate at an outside layer of the missing macroblock 20, and are neighboring to the boundary blocks having correct image data. In the current embodiment, only the missing blocks A00, A01, A10, A11 are illustrated, and the other missing blocks can be learnt by referring to the illustration of the missing blocks A00, A01, A10, A11, according to the symmetrical relation between the missing blocks.
The missing block A00 is neighboring to the correct boundary blocks BB1 and BB16, and an AEBME00 value can be obtained according to equation (11):
                                          A            ⁢                                                  ⁢            E            ⁢                                                  ⁢            B            ⁢                                                  ⁢            M            ⁢                                                  ⁢                          E              00                                =                                                    ∑                                  i                  =                  1                                M                            ⁢                                                          ⁢                                                ∑                                      x                    =                                          x                      0                                                                                                  x                      0                                        +                    N                    -                    1                                                  ⁢                                                                  ⁢                                                                                                                      A                        00                                            ⁡                                              (                                                  x                          ,                                                                                    y                              0                                                        +                            i                                                                          )                                                              -                                          R                      ⁡                                              (                                                                              x                            +                            v                                                    ,                                                                                    y                              0                                                        +                            i                            +                            u                                                                          )                                                                                                                                    +                                          ∑                                  i                  =                  1                                M                            ⁢                                                          ⁢                                                ∑                                      y                    =                                          y                      0                                                                                                  y                      0                                        -                                          (                                              N                        -                        1                                            )                                                                      ⁢                                                                  ⁢                                                                                                                      A                        00                                            ⁡                                              (                                                                                                            x                              0                                                        -                            i                                                    ,                          y                                                )                                                              -                                          R                      ⁡                                              (                                                                                                            x                              0                                                        -                            i                            +                            v                                                    ,                                                      y                            +                            u                                                                          )                                                                                                                                                  ,                            (        11        )            where A00(x, y) represents the pixels of the missing block A00 in the present frame, R(x, y) represents the pixels of the previous frame, (x0, y0) is the coordinate of the missing block A00, (v, u) represents the spatial or temporal motion vector, N represents a quantity of the missing blocks (i.e., 4 in the current embodiment), and M represents a pixel width of the external boundary (i.e., 4 pixels in the current embodiment).
The missing block A10 is neighboring to the boundary blocks BB15, and an AEBME10 value can be obtained according to equation (12):
                                          A            ⁢                                                  ⁢            E            ⁢                                                  ⁢            B            ⁢                                                  ⁢            M            ⁢                                                  ⁢                          E              10                                =                                    ∑                              i                =                1                            M                        ⁢                                                  ⁢                                          ∑                                  y                  =                                      y                    0                                                                                        y                    0                                    -                                      (                                          N                      -                      1                                        )                                                              ⁢                                                          ⁢                                                                                                                                                                                        A                            10                                                    ⁡                                                      (                                                                                                                            x                                  0                                                                -                                i                                                            ,                              y                                                        )                                                                          -                                                                                                                                                R                        ⁡                                                  (                                                                                                                    x                                0                                                            -                              i                              +                              v                                                        ,                                                          y                              +                              u                                                                                )                                                                                                                                                                        ,                            (        12        )            where A10(x, y) represents the pixels of the missing block A10 in the present frame. Similarly, an AEBME01 value of the missing block A01 neighboring to the boundary block BB2 can be obtained in like manner.
The missing blocks A11, B11, C11, D11 in the inside layer are not neighboring to any blocks having correct image data, and therefore they are determined with candidate motion vectors by majority decision according to missing blocks adjacent thereto.
At step S140, according to a complexity of each candidate motion vector of the missing macroblock, the missing blocks are divided into a plurality of sub-macroblocks having optimal sizes, e.g., sub-macroblocks having 16×16, 16×8, 8×16, or 8×8 pixels as shown in FIG. 13, in which T1 represents 16×16 sub-macroblocks, T2 represents 16×8 sub-macroblocks, T3 represents 8×16 sub-macroblocks, and T4 represents 8×8 sub-macroblocks. According to the Laplacian distribution of the motion field constituted by the candidate motion vectors, one having the maximum probability is selected from the T1, T2, T3, and T4 sub-macroblocks by Bayes principle. If the missing macroblock has a relatively regular motion, then larger sub-macroblocks, e.g., sub-macroblocks of 16×16 pixels, are more suitable for subsequent motion refinement. On the contrary, if the missing macroblock has a relatively complex motion, the smaller sub-macroblocks, e.g., sub-macroblocks of 8×8 pixels, are more suitable for subsequent motion refinement.
The sub-macroblock having the maximum probability can be determined by equation (13):
                                                                                          T                  ^                                =                                ⁢                                  max                  ⁢                                      {                                          P                      ⁡                                              (                                                                              T                            j                                                    ❘                          mv                                                )                                                              }                                                              ,                                                j                  ∈                                      {                                          1                      ,                      2                      ,                      3                      ,                      4                                        }                                                  =                                  max                  ⁢                                      {                                                                  P                        ⁡                                                  (                                                                                    T                              j                                                        ⋂                            mv                                                    )                                                                                            P                        ⁡                                                  (                          mv                          )                                                                                      }                                                                                                                                          =                                ⁢                                  max                  ⁢                                      {                                                                                            P                          ⁡                                                      (                                                          mv                              ❘                                                              T                                j                                                                                      )                                                                          ⁢                                                  P                          ⁡                                                      (                                                          T                              j                                                        )                                                                                                                      P                        ⁡                                                  (                          mv                          )                                                                                      }                                                              ,                                                          (        13        )            where P(Tj) represents a probability of a sub-macroblock Tj, and mv represents a motion vector motion vector.
Equation (13) can be further expressed in a form of equation (14):
                                                                                          T                  ^                                ′                            =                            ⁢                              max                ⁢                                  {                                      P                    ⁡                                          (                                              mv                        ❘                                                  T                          j                                                                    )                                                        }                                                                                                        =                            ⁢                              max                ⁢                                  {                                                            ∏                                              k                        =                        1                                                                    N                        j                                                              ⁢                                          P                      ⁡                                              (                                                                              mv                            k                                                    ❘                                                      B                                                          k                              ,                              j                                                                                                      )                                                                              }                                                                                                                        =                                ⁢                                  max                  ⁢                                      {                                                                  ∏                                                  k                          =                          1                                                                          N                          j                                                                    ⁢                                                                                          ⁢                                              [                                                                              ∏                                                                                          mv                                k                                                            ∈                                                              B                                                                  k                                  ,                                  j                                                                                                                                              ⁢                                                                                                          ⁢                                                                                    P                              ⁡                                                              (                                                                                                      mv                                    k                                    x                                                                    ❘                                                                      B                                                                          k                                      ,                                      j                                                                                                                                      )                                                                                      ·                                                          P                              ⁡                                                              (                                                                                                      mv                                    k                                    y                                                                    ❘                                                                      B                                                                          k                                      ,                                      j                                                                                                                                      )                                                                                                                                    ]                                                              }                                                              ,                                                          (        14        )            where Bk,j represents the kth partition in the sub-macroblock Tj, mvk represents the motion vectors of the partition Bk, mvkx represents a horizontal component of the motion vector (mvk), mvky represents a vertical component of the motion vector (mvk), Nj represents the number of partitions in the missing macroblock, where Nj=1 corresponding to T1, Nj=2 corresponding to T2, Nj=2 corresponding to T3, and Nj=4 corresponding to T4.
According to equation (14), equation (15) can be obtained based on the Laplacian distribution of the motion field:
                                                                                          T                  ^                                ′                            =                            ⁢                              max                ⁢                                  {                                                            ∏                                              k                        =                        1                                                                    N                        j                                                              ⁢                                                                                                              ⁢                                                                                                            ⁢                                          [                                                                        ∏                                                                                    mv                              k                                                        ∈                                                          B                                                              k                                ,                                j                                                                                                                                    ⁢                                                                                                  ⁢                                                                              P                            ⁡                                                          (                                                                                                mv                                  k                                  x                                                                ❘                                                                  B                                                                      k                                    ,                                    j                                                                                                                              )                                                                                ·                                                      P                            ⁡                                                          (                                                                                                mv                                  k                                  y                                                                ❘                                                                  B                                                                      k                                    ,                                    j                                                                                                                              )                                                                                                                          ]                                                        }                                                                                                                        =                                ⁢                                  max                  ⁢                                      {                                                                  ∏                                                  k                          =                          1                                                                          N                          j                                                                    ⁢                                                                                          ⁢                                              [                                                                                                                                                                              ∏                                                                                                            mv                                      k                                                                        ∈                                                                          B                                                                              k                                        ,                                        j                                                                                                                                                                            ⁢                                                                                                                                  ⁢                                                                                                      1                                                                          2                                      ⁢                                                                                                                                                          ⁢                                                                              σ                                        k                                        x                                                                                                                                              ⁢                                                                                                            exp                                      ⁡                                                                              (                                                                                                                              -                                                                                                                                                                                                                                          mv                                                  k                                                  x                                                                                                -                                                                                                  μ                                                  k                                                  x                                                                                                                                                                                                                                                                                                                        σ                                            k                                            x                                                                                                                          )                                                                                                              ·                                                                                                                                                                                                                                                                                                            1                                                                      2                                    ⁢                                                                                                                                                  ⁢                                                                          σ                                      k                                      y                                                                                                                                      ⁢                                                                  exp                                  ⁡                                                                      (                                                                                                                  -                                                                                                                                                                                                                      mv                                              k                                              y                                                                                        -                                                                                          μ                                              k                                              y                                                                                                                                                                                                                                                                                            σ                                        k                                        y                                                                                                              )                                                                                                                                                                                                      ]                                                              }                                                              ,                                                          (        15        )            where μkx represents the average horizontal component, μky represents the average vertical component, while σkx and σky can be expressed by:
                    σ        k        x            =                        1                                    2              ⁢                                                          ⁢              a                                      ⁢                              ∑                                          mv                k                            ∈                              B                                  k                  ,                  j                                                              ⁢                                                                mv                k                x                            -                              μ                k                x                                                                      ;                  μ        k        x            =              median        ⁢                  {                      mv            k            x                    }                                        σ        k        y            =                        1                                    2              ⁢                                                          ⁢              a                                      ⁢                              ∑                                          mv                k                            ∈                              B                                  k                  ,                  j                                                              ⁢                                                                mv                k                y                            -                              μ                k                y                                                                      ;                  μ        k        y            =              median        ⁢                              {                          mv              k              y                        }                    .                    When taking a natural log to the above equation, equation (16) is obtained as following:
            T      ^        ′    ≈            max      j        ⁢                  {                              ∑                          k              =              1                                      N              j                                ⁢                                          ⁢                      (                                                            -                  aln                                ⁢                                                                  ⁢                                  σ                  k                  x                                            -                              aln                ⁢                                                                  ⁢                                  σ                  k                  y                                                      )                          }            .      
At step S150, motion refinements are applied to the motion vectors of the sub-macroblocks of the missing macroblock. An adaptive search range (ASR) is obtained according to a first adjacent external boundary matching error (AEBME) at the starting point of a spiral scanning path, along with all of the motion vector values. If the ASR is equal to or less than a threshold value, the motion vector is taken as a motion vector of the missing macroblock, and the threshold value is determined by a proximity of the AEBME value to an original residual value of a sum of absolute differences (SAD).
The ASR is determined by equation (17) according to the AEBME:
                                          A            ⁢                                                  ⁢            S            ⁢                                                  ⁢            R                    =                                                                                                                                  A                      ⁢                                                                                          ⁢                      E                      ⁢                                                                                          ⁢                      B                      ⁢                                                                                          ⁢                      M                      ⁢                                                                                          ⁢                                              E                        ⁡                                                  (                                                      u                            ,                            v                                                    )                                                                                      -                    μ                                                                    μ                            ×                              max                ⁡                                  (                                                            D                      x                                        ,                                          D                      y                                                        )                                            ×                              W                λ                                      +            ɛ                          ,                            (        17        )            where AEBME(u, v) represents the first adjacent external boundary matching error at the starting point, μ represents the original residual values of the SAD, W represents the maximum search size, and λ and ε are constant factors (λ=32 and ε=0). Further, DX and DY can be expressed by equation (18):DX=max|X(MVs)−u|DY=max|Y(MVs)−v|  (18),where X(MVs) represents the X component of the motion vector, and Y(MVs) represents the Y component of the motion vector.
The dynamic threshold (DTα) is expressed by equation (19):DTa=μ+N×α  (19),where N is the total number of pixels, and α is a constant factor for each pixel.
Generally, μ is unavailable, and therefore α is needed in addition for calculating the dynamic threshold (DTα).
FIG. 14 is a schematic diagram illustrating a spiral scanning path according to a conventional technology. Referring to FIG. 14, the spiral scanning path starts from a starting point marked with 0 (in the current embodiment, the starting point is an average motion vector of the candidate motion vectors), in sequence along indication of the arrows, to sequentially perform refinement, so as to search for the best motion vector. For example, the motion vector shown in FIG. 14 marked with 2 represents a desired motion vector, and therefore the motion refinement process processes only three motion vectors. The rest motion vectors are not to be further processed, and therefore the total computation for searching comparison process can be saved. The spiral scanning path saves processing time.
Finally, at step S180, the image data of the missing macroblock can be obtained by moving the candidate macroblock to the position of the missing macroblock according to the motion vector, and thus concealing the negative affection caused by the missing macroblock to the entire image frame.
More details of the conventional TEC methods, ordinary SEC methods and intra high-speed SEC method can be learnt by referring to the publication “Optimization of Hybridized Error Concealment for H.264”, IEEE TRANSACTIONS ON BROADCASTING, VOL. 54, NO. 3, SEPTEMBER 2008.
Although having an efficiency higher than the SEC method, the TEC method cannot be applied when the scenario changes, the image moves fast, the image moves irregularly, or an object appears or disappears, because there is no correct frame having temporal correlation available for reference. As such, a method of an efficient adaptive mode selection for H.264/AVC-coded video delivery in burst-packet-loss networks, and a hybridized error concealment method are highly desired, of improving the overall decoding efficiency and performance.